Search Results: "ashley"

31 August 2009

David Pashley: Setting up gitosis on Jaunty

While git is a completely distributed revision control system, sometimes the lack of a central canonical repository can be annoying. For example, you might want to make your repository published publically, so other people can fork your code, or you might want all your developers to push into (or have code pulled into) a central "golden" tree, that you then use for automated building and continuous integration. This entry should explain how to get this all working on Ubuntu 9.04 (Jaunty). Gitosis is a very useful git repository manager, which adds support like ACLs in pre-commits and gitweb and git-daemon management. While it's possible to set all these things up by hand, gitosis does everything for you. It is nicely configured via git; to make configuration changes, you push the config file changes into gitosis repository on the server. Gitosis is available in Jaunty, but unfortunately there is a bug in the version in Jaunty, which means it doesn't work out of the box. Fortunately there is a fixed version in jaunty-proposed that fixes the main problem. This does mean that you need to add the following to your sources.list:
deb http://gb.archive.ubuntu.com/ubuntu/ jaunty-proposed universe
Run apt-get update && apt-get install gitosis. You should install 0.2+20080825-2ubuntu0.1 or later. There is another small bug in the current version too, as a result of git removing the git-$command scripts out of /usr/bin. Edit /usr/share/python-support/gitosis/gitosis/templates/admin/hooks/post-update and replace
git-update-server-info
with
git update-server-info
With these changes in place, we can now set up our gitosis repository. On the server you are going to use to host your central repositories, run:
sudo -H -u gitosis gitosis-init < id_rsa.pub
The id_rsa.pub file is a public ssh key. As I mentioned, gitosis is managed over git, so you need an initial user to clone and then push changes back into the gitosis repo, so make sure this key belongs to a keypair you have available to the remote user you're going to configure gitosis. Now, on your local computer, you can clone the gitosis-admin repo using:
git clone gitosis@gitserver.example.com:gitosis-admin.git
If you look inside the gitosis-admin directory, you should find a file called gitosis.conf and a directory called keydir. The directory is where you can add ssh public keys for your users. The file is the configuration file for gitosis.
[gitosis]
loglevel = INFO
[group gitosis-admin]
writable = gitosis-admin
members = david@david
[group developers]
members = david@david
writable = publicproject privateproject
[group contributors]
members = george@wilber
writable = publicproject
[repo publicproject]
daemon = yes
gitweb = yes
[repo privateproject]
daemon = no
gitweb = no
This sets up two repositories, called publicproject and privateproject. It enables the public project to be available via the git protocol and in gitweb if you have that installed. We also create two groups, developers and contributors. David has access to both projects, but George only has access to change the publicproject. David can also modify the gitosis configuration. The users are the names of ssh keys (the last part of the line in id_dsa.pub or id_rsa.pub). Once you've changed this file, you can run git add gitosis.conf to add it to the commit, git commit -m "update gitosis configuration to commit it to your local repository, and finally git push to push your commits back up into the central repository. Gitosis should now update the configuration on the server to match the config file. One last thing to do is to enable git-daemon, so people can anonymously clone your projects. Create /etc/event.d/git-daemon with the following contents:
start on startup
stop on shutdown
exec /usr/bin/git daemon \
   --user=gitosis --group=gitosis \
   --user-path=public-git \
   --verbose \
   --syslog \
   --reuseaddr \
   --base-path=/srv/gitosis/repositories/ 
respawn
You can now start this using start git-daemon So now, you need to start using your repository. You can either start with an existing project or an empty directory. Start by running git init and then git add $file to add each of the files you want in your project, and finally git commit to commit them to your local repository. The final task is to add a remote repository and push your code into it.
git remote add origin gitosis@gitserver.example.com:privateproject.git
git push origin master:refs/heads/master
In future, you should be able to do git push to push your changes back into the central repository. You can also clone a project using git or ssh, providing you have access, using the following commands. The first is for read-write access over ssh and the second uses the git protocol for read-only access. The git protocol uses TCP port 9418, so make sure that's available externally, if you want the world to be able to clone your repos.
git clone gitosis@gitserver.example.com:publicproject.git
git clone git://gitserver.example.com/publicproject.git
Setting up GitWeb is left as an exercise for the reader (and myself because I am yet to attempt to set that up).
Read Comments (0)

10 April 2009

David Pashley: Check maven dependencies

One really nice feature of maven is the dependency resolution stuff that it does. The dependency plugin also has an analyse goal that can detect a number of problems with your dependencies. It can detect libraries you use but haven't declared in your POM, but work through transitive dependencies. This can cause build problems when you remove the library that was dragging in the undeclared dependency. It can also work out which dependencies you are no longer using, but have a declared dependency.
mojo-jojo david% mvn dependency:analyze
[INFO] Scanning for projects...
...
[INFO] [dependency:analyze]
[WARNING] Used undeclared dependencies found:
[WARNING]    commons-collections:commons-collections:jar:3.2:compile
[WARNING]    commons-validator:commons-validator:jar:1.3.1:compile
[WARNING]    org.apache.myfaces.core:myfaces-api:jar:1.2.6:compile
[WARNING] Unused declared dependencies found:
[WARNING]    javax.faces:jsf-api:jar:1.2_02:compile
...
Read Comments (0)

David Pashley: How not to configure your DNS

How not to configure your DNS
david% dig -x 190.208.19.230
; <<>> DiG 9.4.2-P2 <<>> -x 190.208.19.230
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35398
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;230.19.208.190.in-addr.arpa.   IN      PTR
;; ANSWER SECTION:
230.19.208.190.in-addr.arpa. 3600 IN    PTR     190.208.19.230.
;; Query time: 253 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Apr 10 10:00:21 2009
;; MSG SIZE  rcvd: 73
Whoops
Read Comments (1)

14 March 2009

David Pashley: Fast Servlet development with Maven and Jetty

On off the biggest problems with developing servlets under a container like Tomcat is the amount of time taken to build your code, deploy it to the container and restart it to pick up any changes. Maven and the Jetty plugin allow you to cut down on this cycle considerably. The first step is to allow you to start your application in maven by running:
mvn jetty:run
We do this by configuring the jetty plugin inside our pom.xml:
<plugin>
   <groupId>org.mortbay.jetty</groupId>
   <artifactId>maven-jetty-plugin</artifactId>
   <version>6.1.10</version>
</plugin>
Now when you run mvn jetty:run your application will start up. But we can improve on this. The Jetty plugin can be configured to scan your project every so often and rebuild it and reload it if anything changes. We do this by changing our pom.xml to read:
<plugin>
   <groupId>org.mortbay.jetty</groupId>
   <artifactId>maven-jetty-plugin</artifactId>
   <version>6.1.10</version>
   <configuration>
      <scanIntervalSeconds>10</scanIntervalSeconds>
   </configuration>
</plugin>
Now when you save a file in your IDE, by the time you've switched to your web browser, Jetty is already running your updated code. Your development cycle is almost up to the same speed as Perl or PHP. You can find more information at the plugin page.
Read Comments (1)

5 March 2009

Martin F. Krafft: Don't get fooled by the cost-average effect

I have a bit of money invested in stocks and funds, passively managed by my bank. I ve made a bit with risky speculations, and I ve lost about the same amount with the markets dive of the last 18 months. Net benefit: zero cash, a bit of experience. Time to move on I am not an interesting customer, so I was all the more surprised to get a call from an employee of the bank the other day. He identified himself as Crisis Manager or something similarly ridiculous, mentioned at least 20 times that he was a regular employee and wouldn t receive provisions, and he also kept assuring me that he s doing himself what he s suggesting to me: leveraging the cost-average effect. I turned him down, probably a bit too briskily. Let s investigate why. Banks are good at devising ingenious tricks to fool people; it s their business after all. One of those tricks is the so-called cost-average effect. It s awesome, they say, because it is the first product that adapts to the ever-changing market: when prices are up, it buys smaller numbers of shares, and in low times, it stocks up on them by buying larger quantities, automatically. Fancy, no? It isn t, because the entire idea is to get your signature on a contract that obliges you to pay a fixed sum each month, with which shares are then bought. Obviously, when prices are up, numbers you get for your money are low, and vice versa. Observe how well the banks manage to turn that basic rule into a selling point. They are good at that. In fact, that s probably all they re good at. Obviously , inflation is also factored in, so the monthly fixed sum actually increases by a projected 5% each year, which they call inflation protection . While this strategy investing fixed-sum installments rather than lump sums certainly has its benefits, and can level out the risk of throwing a large sum of money at the market, it is commonly considered a suboptimal strategy, especially when employed over longer spans of time. For more information, I refer you to the frequently cited article Nobody gains from dollar cost averaging analytical, numerical and empirical results , published in the 1992/1993 issue of the Financial Services Review journal. So why would the bank approach me with this offer ? Let s go hypothetical and check out Ashley s investment practices: Ashley bought a fund five years ago at 100 per share, and it s now down to 80 . The loss is thus 20%, a -4% factor (20% / 5 years) that affects the overall performance of the portfolio. Ashley doesn t really care much about the percentages. After all, the only thing that counts is the worth of the shares when sold right now. Everything else is history and only influences the size of the tears one weeps in times like this, which is a psychological malfunction, not a function of the financial markets crisis. The bank, however, does not like negative performance values. Positive percentages yield happy (blinded?) customers, who get much more enthusiastic (greedy?) to put more money into the giant machine that makes the banks tick which is still in the process of blowing up into their faces. But negative performance values cause investors to become conservative and leery. They d rather hold on to their hard-earned cash for fear of losing more. That s the flip-side of aforementioned psychological malfunction, and this is the core of the reason why some of the big investors get bigger even in times like this they do not succomb to the feeling . The banks, however, need cash, and loads of it, and while the state is busy printing notes, they take every chance to get more. Private clients are wonderful prey, as they are less informed about the system than investors. So, to get cash from the clients, the banks have to circumvent the psychological malfunction. They cannot boost the market value share, but there are two values that factor into a percentage, and the cost-average effect, sold as the best thing since sliced bread, is nothing but a way to affect the other number: the sale price, the divisor. If the fund stays below the original sale price (100 ), each time Ashley pumps more cash into the machinery each month, this divisor decreases, effectively decreasing the percentage value. In the long run, this yields happier customers, who are willing to put more money in, especially if the performance turns positive. The baseline is now lower, so a return to previous heights would correspond to larger performance values than ever before. Obviously, a positive market causes the baseline to grow, but the cost-average deals continues to pay off for the banks. Since Ashley put that signature on the contract, the bank has a guaranteed cash flow for the contract s duration, while Ashley wonders why the percentage remains dampened. Cost-average means decreasing the percentage, in good as well as in bad times. Just now, the bank doesn t have to work so hard to get even more money from you. It gets worse: usually, those contracts are embedded in some sort of life insurance deals with grand promises in 40 years to come, and over all that time, it ll be quite clear who wins and who loses. Scary, huh? I was going to end with the advice to stay away from the cost-average effect, but I think I can just as well make it more general: if you don t understand a deal, don t sign it. If you abide by that rule, you would stay away from cost-average effect deals all by yourself. Maybe my little writeup has helped, nonetheless. NP: DJ Ti sto: Live at Innercity

15 February 2009

David Pashley: MySQL silently truncating your data: Update

After my entry yesterday about MySQL truncating data, several people have pointed out that MySQL 4.1 or later gives you a warning. Yes, this is true. You can even see it in the example I gave:
Query OK, 1 row affected, 1 warning (0.00 sec)
I ignored mentioning this, but perhaps should have said something about it. I reason I didn't mention it was because I didn't feel that a warning really helped anyone. Developers have enough problems remembering to check for errors, let along remembering to check in case there was a warning as well. Plus, they'd then have to work out if the warning was something serious or something they could ignore. There's also the question of how well the language bindings present this information. Take for example, PHP. The mysqli extension gained support for checking for warnings in PHP5 and gives the following code as an example of getting warnings:
mysqli_query($link, $query);
if (mysqli_warning_count($link))  
   if ($result = mysqli_query($link, "SHOW WARNINGS"))  
      $row = mysqli_fetch_row($result);
      printf("%s (%d): %s\n", $row[0], $row[1], $row[2]);
      mysqli_free_result($result);
    
 
Hardly concise code. As of 5.1.0, there is also mysqli_get_warnings(), but is undocumented beyond noting its existence. The MySQL extension does not support getting warning information. The PDO wrapper doesn't provide any way to get this information. In perl, DBD::mysql has a mysql_warning_count() function, but presumably would have to call "SHOW WARNINGS" like in the PHP example. Seems Python's MYSQLdb module will raise an exception on warnings in certain cases. Mostly using the Cursor object. In java, you can set the jdbcCompliantTruncation connection parameter to make the driver throw java.sql.DataTruncation exceptions, as per the JDBC spec, which makes you wonder why this isn't set by default. Unfortunately this setting is usually outside the programmer's control. There is also the java.sql.Statement.getWarnings(), but once again, you need to check this after every statement. Not sure if ORM tools like Hibernate check this or not. So, yes MySQL does give you a warning, but in practice is useless.
Read Comments (0)

David Pashley: MySQL silently truncating your data

MySQL in its standard configuration has this wonderful "feature" of truncating your data if it can't fit in the field.
mysql> create table foo (bar varchar(4));
Query OK, 0 rows affected (0.00 sec)
mysql> insert into foo (bar) values ("12345");
Query OK, 1 row affected, 1 warning (0.00 sec)
In comparison, PostgeSQL does:
psql=> create table foo (bar varchar(4));
CREATE TABLE
psql=> insert into foo (bar) values ('12345');
ERROR:  value too long for type character varying(4)
You can make MySQL do the right thing by setting the SQL Mode option to include STRICT_TRANS_TABLES or STRICT_ALL_TABLES. The difference is that the former will only enable it for transactional data storage engines. As much as I'm loathed to say it, I don't recommend using STRICT_ALL_TABLES, as an error during updating a non-transational table will result in a partial update, which is probably worse than a truncated field. Setting the mode to TRADITIONAL includes both these and a couple of related issues (NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO) You can set the mode using: Just say no to databases that happily throw away your data
Read Comments (4)

25 January 2009

David Pashley: Subversion and "(502 Bad Gateway) in response to COPY request" errors

Was attempting to merge a branch in one of my projects and upon committing the merge, I kept getting this error:
mojo-jojo david% svn commit -m "merge in the maven branch"
Sending        trunk
Sending        trunk/.classpath
Sending        trunk/.project
Adding         trunk/.settings
svn: Commit failed (details follow):
svn: Server sent unexpected return value (502 Bad Gateway) in response
to COPY request for '/svn/eddie/!svn/bc/314/branches/maven/.settings'
A quick search found several other people having the same problem. Seems it only happens for https repositories using mod_dav_svn. The solution is to make sure that your virtual host in apache has explicit SSL config options, even if you are using an SSL config from a default virtual host. For example, I added the following to my subversion vhost, which was just copied from my default vhost:
SSLEngine on
SSLCertificateFile /etc/apache2/ssl/catnip.org.uk.crt
SSLCertificateKeyFile /etc/apache2/ssl/catnip.org.uk.key
Read Comments (0)

21 January 2009

David Pashley: DBI boilerplate code

I keep writing code to talk to databases in perl and I'm forever forgetting the correct runes for talking to databases, so I thought I'd stick it here for easy reference.
use DBI;
my $db_driver = "Pg" # Pg or mysql (or others)
my $db_name = "database";
my $db_host = "localhost";
my $db_user = "username";
my $db_pass = "password";
my $dbh = DBI->connect("dbi:$db_driver:dbname=$db_name;host=$db_host", 
   $db_user, $db_pass);
It's probably handy to give an example of a common database read operation
my $sth = $dbh->prepare( "SELECT * FROM table WHERE id  = ?") 
      or die $dbh->errstr;
$sth->execute($id) or die $dbh->errstr;
while (my $hashref = $sth->fetchrow_hashref)  
   print $hashref-> id ;
 
Read Comments (1)

26 November 2008

David Pashley: Vim Syntax Highlighting for Puppet

I've just set up syntax highlighting for Puppet manifest files, and thought I'd share the simple steps. The first thing to do is download the syntax file from http://www.reductivelabs.com/downloads/puppet/puppet.vim and save this to ~/.vim/syntax/puppet.vim. Now when the filetype is set to "puppet", vim will use this syntax file. That's useful, it it would be even nicer if we could make vim know that files ending in .pp were puppet files. Turns out this is very easy to do. You need to create a file to detect the correct filetype when you open a file. You need to put the following lines in ~/.vim/ftdetect/puppet.vim:
au BufRead,BufNewFile *.pp   setfiletype puppet
Now when you load a file ending in .pp, you should get nice syntax highlighting. You can also make vim use special settings for the puppet filetype by creating a vim script file in one of ~/.vim/ftplugin/puppet.vim, ~/.vim/ftplugin/puppet_*.vim and/or ~/.vim/ftplugin/puppet/*.vim. Vim has a lot of flexible hooks to enable file type specific configuration; hopefully it should be fairly easy to modify these examples for other file formats.
Read Comments (0)

24 October 2008

David Pashley: Setting up Ubuntu PXE booting

I've recently had to set up a new machine, but didn't have an install cdrom available, so I decided to use the easiest method for installing Ubuntu; PXE booting. Here's how I did it. PXE involves setting up two simple technologies, DHCP and TFTP. We start by setting up TFTP. TFTP is Trivial File Transfer Protocol, a cut down version of FTP. There are a number of TFTP servers in Debian and Ubuntu, but not all of them support the extensions that the pxelinux bootloader used by debian-installer need. Experience has shown that tftpd-hpa works correctly, so we'll want to install that.
ace root% apt-get install tftpd-hpa
Note: If this installs an inetd at the same time, you may need to restart the inetd so it enables the tftpd service. The tftpd will serve files out of /var/lib/tftpboot, so we need to add some files for it to serve. You can use this script to fetch various netboot installers from Ubuntu's servers.
#!/bin/bash
set -u
set -e
cd /var/lib/tftpboot
for dist in dapper feisty gutsy hardy intrepid; do
    mkdir -p $dist
    for arch in amd64 i386; do
        mkdir -p $dist/$arch/
        (cd $dist/$arch/ && ncftpget -RT \
           ftp://archive.ubuntu.com/ubuntu/dists/$dist/main/installer-$arch/current/images/netboot/)
    done
done
Download ubuntu-tftp-update.sh Now we need to alter our dhcpd configuration. (You are using DHCP aren't you?) All we need to add is a group declaration to your subnet declaration, adding a next-server and a filename parameter. You can then add a host declaration for any machine you want to netboot into the installer.
group   # intrepid amd64
     next-server 10.0.0.1;
     filename "intrepid/amd64/pxelinux.0";
     host foobar   hardware ethernet 00:22:15:45:cc:fa; fixed-address foobar.example.com;  
 
You'll need to restart the dhcp server so it picks up the new setting. The next-server parameter is the name or IP address of your tftp server. filename is the path to the bootloader. Obviously, you can use this to pick which version of the installer you want to run. If you do a lot of installations, it might be worth configuring every installer you're likely to use and then move hosts in and out of the suitable group as and when you need to install them. All that's left to do now is to boot the computer and set it to boot from the network and enjoy medialess installation.
Read Comments (1)

2 October 2008

David Pashley: Slaves *and* Caching

Dear Lazyweb, We have a web application that has quite a large database and reasonable usage. Back in the dim and distant past, we scaled the application by the age-old method of using several read-only slave databases to prevent reads on the master swamping writes. This worked well for several years, and then we introduced memcached into the mix to improve performance by reducing the number of reads from the database. This improved our database capacity even further. Now the question has arisen about reducing or even removing the code to read from the slaves. I'm trying to come up with some compelling reasons to keep the application reading from the slaves. The pros and cons I currently have for removing the code are:
Pros
  • Reduces code complexity
  • Removes consistency problems due to latency in the replication. This is less of a problem than it used to be after we solved a problem with our replication
Cons
  • Reduces our existing capacity
  • Cache flushes would cause huge spikes on our master server until the cache filled up again
  • Caches wouldn't help queries with unique critera
I would appreciate any additional reasons, pro or cons. We already have an existing non-live slave for backups and slow queries by developers. We would retain a slave for redundancy in the case of master failure. I'm only looking for issues that would affect the application.
Read Comments (0)

20 August 2008

David Pashley: Asymmetric Routing and Flow Sessions in JUNOS ES

We've recently installed a couple of Juniper J-Series routers that have the new JUNOS with Enhanced Services installed on them. During the transition from our existing Linux routers, we started moving internal subnets to the new routers, but when we moved the first subnet, we discovered a problem with hosts that had addresses on two different subnets. Connections would either connect for a minute and work and then get a connection reset, or packets would come in to the server and leave again, but then get swallowed in the ether. I spent quite a bit of time this week reading about the security features of the new routers, and finally came up with a solution. The first clue was that I was getting connection reset from something on the network, but carrying out packet sniffing on our existing routers and the end points showed that they weren't generating it. I eventually found the tcp-rst option, which generates a reset packet for any non-SYN packet that doesn't match an existing flow session. JUNOS ES does stateful packet inspection by creating a session when it sees an initial SYN packet and then does filtering and routing based on that flow session so it doesn't have to do it for every packet. When I turned off the tcp-rst option on the trust zone, my connection that worked for a minute worked again only for a minute, but this time, it just hung, rather than dying with a connection reset. This cemented the idea that the Juniper routers were the cause. It turned out that the problem was that there was asynchronous routing going on. A packet was coming in to 10.0.0.1/24, but the server was also on 10.0.1.1/24 and the default route nexthop was 10.0.1.2. Depending on which subnet we moved depended on the resulting behavour. If we moved 10.0.0.0/24 to the new routers, they would only see the incoming side of the conversation. If we moved 10.0.1.0/24, they would only see the outgoing side of the conversation. If we then think how this would work with the session-based routing and firewalling, in the first case, the router would see the initial SYN packet, but would never see the returning SYN-ACK packet, and after an initial timeout, decide the flow never established and destroy the session info, resulting in further incoming packets to be dropped. In the second case, it would never see the SYN packet, only the SYN-ACK. This packet wouldn't belong to an existing session, so would be blocked. The solution is to turn off the SYN check, using:
[edit]
user@host# set security flow tcp-session no-syn-check
After commiting that, sessions work correctly, even without the router seeing both sides of the connection:
user@host> show security flow session destination-prefix 10.0.0.1    
Session ID: 201341, Policy name: default-permit/4, Timeout: 1798
  In: 192.168.0.1/61136 --> 10.0.0.1/22;tcp, If: ge-0/0/0.7
  Out: 10.0.0.1/22 --> 192.168.0.1/61136;tcp, If: ge-0/0/1.7
1 sessions displayed
Sadly, I couldn't find much information on the no-syn-check option, so hopefully people will find this explaination useful.
Read Comments (1)

12 July 2008

David Pashley: Rebuilding a RAID array

I recently had a failed drive in my RAID1 array. I've just installed the replacement drive and thought I'd share the method. Let's look at the current situation:
root@ace:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 sda3[1]
      483403776 blocks [2/1] [_U]
      
md0 : active raid1 sda1[1]
      96256 blocks [2/1] [_U]
      
unused devices: <none>
So we can see we have two mirrored arrays with one drive missing in both. Let's see that we've recognised the second drive:
root@ace:~# dmesg   grep sd
[   21.465395] Driver 'sd' needs updating - please use bus_type methods
[   21.465486] sd 2:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB)
[   21.465496] sd 2:0:0:0: [sda] Write Protect is off
[   21.465498] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   21.465512] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   21.465562] sd 2:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB)
[   21.465571] sd 2:0:0:0: [sda] Write Protect is off
[   21.465573] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   21.465587] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   21.465590]  sda: sda1 sda2 sda3
[   21.487248] sd 2:0:0:0: [sda] Attached SCSI disk
[   21.487303] sd 2:0:1:0: [sdb] 976773168 512-byte hardware sectors (500108 MB)
[   21.487314] sd 2:0:1:0: [sdb] Write Protect is off
[   21.487317] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[   21.487331] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   21.487371] sd 2:0:1:0: [sdb] 976773168 512-byte hardware sectors (500108 MB)
[   21.487381] sd 2:0:1:0: [sdb] Write Protect is off
[   21.487382] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[   21.487403] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   21.487407]  sdb: unknown partition table
[   21.502763] sd 2:0:1:0: [sdb] Attached SCSI disk
[   21.506690] sd 2:0:0:0: Attached scsi generic sg0 type 0
[   21.506711] sd 2:0:1:0: Attached scsi generic sg1 type 0
[   21.793835] md: bind<sda1>
[   21.858027] md: bind<sda3>
So, sda has three partitions, sda1, sda2 and sda3, and sdb has no partition table. Let's give it one the same as sda. The easiest way to do this is using sfdisk:
root@ace:~# sfdisk -d /dev/sda   sfdisk /dev/sdb
Checking that no-one is using this disk right now ...
OK
Disk /dev/sdb: 60801 cylinders, 255 heads, 63 sectors/track
sfdisk: ERROR: sector 0 does not have an MSDOS signature
 /dev/sdb: unrecognised partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0
   Device Boot    Start       End   #sectors  Id  System
/dev/sdb1   *        63    192779     192717  fd  Linux RAID autodetect
/dev/sdb2        192780   9960299    9767520  82  Linux swap / Solaris
/dev/sdb3       9960300 976768064  966807765  fd  Linux RAID autodetect
/dev/sdb4             0         -          0   0  Empty
Successfully wrote the new partition table
Re-reading the partition table ...
If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
If we check dmesg now to check it's worked, we'll see:
root@ace:~# dmesg   grep sd
...
[  224.246102] sd 2:0:1:0: [sdb] 976773168 512-byte hardware sectors (500108 MB)
[  224.246322] sd 2:0:1:0: [sdb] Write Protect is off
[  224.246325] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[  224.246547] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  224.246686]  sdb: unknown partition table
[  227.326278] sd 2:0:1:0: [sdb] 976773168 512-byte hardware sectors (500108 MB)
[  227.326504] sd 2:0:1:0: [sdb] Write Protect is off
[  227.326507] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[  227.326703] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  227.326708]  sdb: sdb1 sdb2 sdb3
So, now we have identical partition tables. The next thing to do is to add the new partitions to the array:
root@ace:~# mdadm /dev/md0 --add /dev/sdb1
mdadm: added /dev/sdb1
root@ace:~# mdadm /dev/md1 --add /dev/sdb3
mdadm: added /dev/sdb3
Everything looks good. Let's check dmesg:
[  323.941542] md: bind<sdb1>
[  324.038183] RAID1 conf printout:
[  324.038189]  --- wd:1 rd:2
[  324.038192]  disk 0, wo:1, o:1, dev:sdb1
[  324.038195]  disk 1, wo:0, o:1, dev:sda1
[  324.038300] md: recovery of RAID array md0
[  324.038303] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  324.038305] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[  324.038310] md: using 128k window, over a total of 96256 blocks.
[  325.417219] md: md0: recovery done.
[  325.453629] RAID1 conf printout:
[  325.453632]  --- wd:2 rd:2
[  325.453634]  disk 0, wo:0, o:1, dev:sdb1
[  325.453636]  disk 1, wo:0, o:1, dev:sda1
[  347.970105] md: bind<sdb3>
[  348.004566] RAID1 conf printout:
[  348.004571]  --- wd:1 rd:2
[  348.004573]  disk 0, wo:1, o:1, dev:sdb3
[  348.004574]  disk 1, wo:0, o:1, dev:sda3
[  348.004657] md: recovery of RAID array md1
[  348.004659] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  348.004660] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[  348.004664] md: using 128k window, over a total of 483403776 blocks.
Everything still looks good. Let's sit back and watch it rebuild using the wonderfully useful watch command:
root@ace:~# watch -n 1 cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 sdb3[2] sda3[1]
      483403776 blocks [2/1] [_U]
      [=====>...............]  recovery = 26.0% (126080960/483403776) finish=96.2min speed=61846K/sec
      
md0 : active raid1 sdb1[0] sda1[1]
      96256 blocks [2/2] [UU]
      
unused devices: <none>
The Ubuntu and Debian installers will allow you create RAID1 arrays with less drives than you actually have, so you can use this technique if you plan to add an additional drive after you've installed the system. Just tell it the eventual number of drives, but only select the available partitions during RAID setup. I used this method when a new machine recent didn't have enough SATA power cables and had to wait for an adaptor to be delivered. (Why did no one tell me about watch until recently. I wonder how many more incredibly useful programs I've not discovered even after 10 years of using Linux)
Read Comments (3)

23 June 2008

MJ Ray: Firefox 3, day 6: security flaw and banks

I didn't spot this when I wrote my last post, but it seems there's a security alert for FF3 already - hackademix.net: Firefox 3 Untimely Security Advisory - but it also affects FF2 and probably my cautious Javascript settings are enough to stop it anyway, looking at that report. I've also been sent another update to the page on Online Banking with GNU/Linux, Firefox-based browsers or Free Software (first direct plus using ActiveX) - I wonder if any bankers will be noticeably slow to allow FF3 and will any of them cite this security flaw? I hope not - UK online banking security is hardly in a good place to throw stones. I was mildly surprised that the list was linked from Ashley Highfield's BBC blog on Testing Linux Ubuntu but I've no idea why he doubts the list's accuracy! It's as accurate as its contributors - most of whom I name - and I'm willing to put my name to it too. That's better than Wikipedia, which the BBC uses far too much IMO. Would he trust the list more if it was anonymously-edited on a public site? Anyway, I guess I should move that list to a more permanent location soon. Previous FF3 parts: Firefox 3, day 3: first impressions and 7 Reasons Why Firefox 3 Download Day Sucks

10 June 2008

David Pashley: index sambaSID sub

If you get the following error:
/etc/ldap/slapd.conf: line 127: substr index of attribute "sambaSID" disallowed
when you run slapindex, then you haven't updated your samba.schema to the version from Samba 3.0.23. Dapper and Edgy had 3.0.22, so if you've recently upgraded to Hardy, you will see this problem. The file should have an MD5 of 0e23b3ad05cd2b38a302fe61c921f300. I'm hoping this resolves problems I have with samba not picking up group membership changes. I'll update if it does. Update: Having installed the new schema and run slapindex, net rpc info shows I have twelve groups when previously it showed zero. This may not solve my group membership problems, but it can't be a step backwards.
Read Comments (0)

9 June 2008

David Pashley: Compiled Regexes in Spamassassin 3.2

Spamassassin 3.2, which is available in Gutsy and Lenny, comes with a new feature to increase performance by compiling its regular expressions using re2c. It's very quick to enable. First, you need to install the required packages:
apt-get install re2c libc6-dev gcc make
Next, edit /etc/spamassassin/v320.pre and uncomment the line that says:
loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody
Next pre-compile the regular expressions using sa-compile:
femme:/etc/logcheck# sa-compile
[18741] info: generic: base extraction starting. this can take a while...
[18741] info: generic: extracting from rules of type body_0
100% [===========================] 3293.83 rules/sec 00m00s DONE
100% [===========================] 650.12 bases/sec 00m01s DONE
[18741] info: body_0: 647 base strings extracted in 2 seconds
[snip compiler output]
make install
Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /var/lib/spamassassin/compiled/3.002004/auto/Mail/SpamAssassin/CompiledRegexps/body_0/body_0.so
Installing /tmp/.spamassassin18741hDrlUQtmp/ignored/man/man3/Mail::SpamAssassin::CompiledRegexps::body_0.3pm
Writing /var/lib/spamassassin/compiled/3.002004/auto/Mail/SpamAssassin/CompiledRegexps/body_0/.packlist
Appending installation info to /var/lib/spamassassin/compiled/3.002004/perllocal.pod
cp /tmp/.spamassassin18741hDrlUQtmp/bases_body_0.pl /var/lib/spamassassin/compiled/3.002004/bases_body_0.pl
cd /
rm -rf /tmp/.spamassassin18741hDrlUQtmp
Finally, restart spamassassin, and you should find it runs faster. You will need to run sa-compile every time you update your rules, or they won't take effect. If you get the following warning:
Can't locate Mail/SpamAssassin/CompiledRegexps/body_0.pm in @INC
you forgot to run sa-compile; re-run it and the error should go away.
Read Comments (0)

22 May 2008

David Pashley: Apache 2.2 auth_ldap config

Apache 2.2 changed the way you configure LDAP authentication. mod_auth_ldap was replaced with mod_authnz_ldap, so don't forget to enable the new module and disable the old one. Because I'll always forget, here's the new style config.
AuthType basic
AuthName "admin"
AuthBasicProvider ldap
AuthLDAPUrl ldap://ldap.example.com:389/ou=people,dc=example,dc=com?uid?sub
AuthLDAPGroupAttributeIsDN off
Require ldap-group cn=systems,ou=groups,dc=example,dc=com
AuthLDAPGroupAttribute memberUid
The sections in bold are the sections I had to change from the 2.0 config.
Read Comments ()

1 May 2008

David Pashley: Not like not like not like

I should have mentioned that my previous blog posting was using MySQL 4.0(4.0.23_Debian-3ubuntu2-log). It seems that in 5.0.2 they changed the precedence of the NOT operator to be lower than LIKE. From the manual:
The precedence shown for NOT is as of MySQL 5.0.2. For earlier versions, or from 5.0.2 on if the HIGH_NOT_PRECEDENCE SQL mode is enabled, the precedence of NOT is the same as that of the ! operator.
Using 5.0 (5.0.22-Debian_0ubuntu6.06.8-log), and a slightly smaller dataset, I get:
mysql> select count(*) from Table where blobid is null or not blobid like '%-%';
+----------+
  count(*)  
+----------+
    199057   
+----------+
1 row in set (3.26 sec)
mysql> select count(*) from Table where blobid is null or blobid not like '%-%';
+----------+
  count(*)  
+----------+
    199057   
+----------+
1 row in set (0.96 sec)
Jim Kingdon experimented with other databases and was unable to reproduce this problem. My test with PostgreSQL 8.3:
quux=> create table foo (blobid varchar(255));
CREATE TABLE
quux=> insert into foo (blobid) values 
   ('5cd1237469cc4b52ca094e215156c582-9ef460ac4134c600a4d2382c4b0acee7'), 
   (NULL), 
   ('d20cb4037f8f9ab1de5de264660f005c-2c34209dcfb39251cf7c16bb6754bbd2'), 
   ('845a8d06719d8bad521455a8dd47745c-095d9a0831433c92cd269e14e717b3a9'),
   ('9580ed23f34dd68d35da82f7b2a293d6-bf39df7509d977a1de767340536ebe80'), 
   ('06c9521472cdac02a2d4b2a18f8bec0f-0a8a28d3b63df54860055f1d1de92969'), 
   ('ed3cd0dd9b55f76db7544eeb64f3cfa0-80a6a3eb6d73c0a58f88b7c332866d5c'),
   (NULL),
   ('b339f6545651fbfa49fa500b7845c4ce-6defb5ffc188b8f72f1aa10bbd5c6bec'),
   ('642075963d6f69bb11c35a110dd07c2c8db54ac2d2accae7fa4a22db1d6caae9');
INSERT 0 10
quux=> select count(*) from foo 
   where blobid is null or blobid not like '%-%';
 count 
-------
     3
(1 row)
quux=> select count(*) from foo 
   where blobid is null or not blobid like '%-%';
 count 
-------
     3
(1 row)
quux=> select not blobid from foo limit 10;
ERROR:  argument of NOT must be type boolean, not type character varying
This appears to have been the case since at least 7.4 Problems like this is going to make the transition from MySQL 4.0 to 5.x all the more fun when we get around to doing it.
Read Comments (0)

30 April 2008

David Pashley: Not like not like not like

Dear lazyweb, I'm possibly being stupid, but can someone explain the differences between these two queries?
mysql> select count(*) from Table 
   where blobid is null or not blobid like '%-%';
+----------+
  count(*)  
+----------+
  15262487  
+----------+
1 row in set (25 min 4.18 sec)
mysql> select count(*) from Table 
   where blobid is null or blobid not like '%-%';
+----------+
  count(*)  
+----------+
  20044216  
+----------+
1 row in set (24 min 54.06 sec)
For reference:
mysql> select count(*) from Table where blobid is null;
+----------+
  count(*)  
+----------+
  15262127  
+----------+
1 row in set (24 min 7.15 sec)
Update: It turns out that the former was doing (not blobid) like '%-%' which turns out to not do anything sensible:
mysql> select not blobid from Table limit 10;
+------------+
  not blobid  
+------------+
           0  
        NULL  
           1  
           0  
           0  
           0  
           1  
        NULL  
           1  
           0  
+------------+
10 rows in set (0.02 sec)
mysql> select blobid from Table limit 10;
+-------------------------------------------------------------------+
  blobid                                                             
+-------------------------------------------------------------------+
  5cd1237469cc4b52ca094e215156c582-9ef460ac4134c600a4d2382c4b0acee7  
  NULL                                                               
  d20cb4037f8f9ab1de5de264660f005c-2c34209dcfb39251cf7c16bb6754bbd2  
  845a8d06719d8bad521455a8dd47745c-095d9a0831433c92cd269e14e717b3a9  
  9580ed23f34dd68d35da82f7b2a293d6-bf39df7509d977a1de767340536ebe80  
  06c9521472cdac02a2d4b2a18f8bec0f-0a8a28d3b63df54860055f1d1de92969  
  ed3cd0dd9b55f76db7544eeb64f3cfa0-80a6a3eb6d73c0a58f88b7c332866d5c  
  NULL                                                               
  b339f6545651fbfa49fa500b7845c4ce-6defb5ffc188b8f72f1aa10bbd5c6bec  
  642075963d6f69bb11c35a110dd07c2c-8db54ac2d2accae7fa4a22db1d6caae9  
+-------------------------------------------------------------------+
10 rows in set (0.00 sec)
The documentation says Logical NOT. Evaluates to 1 if the operand is 0, to 0 if the operand is non-zero, and NOT NULL returns NULL. but doesn't describe the behaviour of NOT 'string'. It would appear that a string starting with a number returns 0 and a string starting with a letter returns 1. Either way, neither has a hyphen in.
Read Comments (2)

Next.

Previous.